Systems and methods for generation of composite video

ABSTRACT

Systems and methods for generating a composite video based on a plurality of input streams are provided. A first video stream is received from a first device. A second video stream and coordination information are received from a second device. The coordination information includes instructions for combining the first video stream and the second video stream in a composite video. The first video stream and the second video stream are time aligned, and the instructions of the coordination information and the time-aligned first video stream and second video stream are used to generate the composite video.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a Continuation of U.S. patent application Ser. No.15/012,640 filed Feb. 1, 2016, which is a Continuation of U.S. patentapplication Ser. No. 14/075,134 filed Nov. 8, 2013, now U.S. Pat. No.9,251,852, which claims priority to U.S. Provisional Patent ApplicationNo. 61/726,851, filed Nov. 15, 2012, entitled “Systems and Methods forGeneration of Composite Video,” the contents of which are hereinincorporated by reference in their entireties.

This application is related to U.S. Provisional Patent Application No.61/641,946, filed May 3, 2012, U.S. patent application Ser. No.11/489,840, filed Jul. 20, 2006, and U.S. patent application Ser. No.11/634,441, filed Dec. 6, 2006, the disclosure of all of which is hereinincorporated by reference.

TECHNICAL FIELD

The technology described herein relates generally to video productionand more particularly to composite video production from multiple inputvideo sources.

BACKGROUND

When displaying a recorded video (e.g., video of a sporting event,academic lecture, or news story), it is often desirable to supplementthe video with related text, images, audio, and/or other video. Forexample, it may be desirable to supplement a recorded video of aprofessor's lecture with presentation slides including related text,images, or video, such that the lecture video and presentation slidesare displayed simultaneously in separate sub-regions of an overalldisplay. However, creating a final video product containing the originalrecorded video and supplemental material may be a time-consuming andexpensive process. Oftentimes, costly video editing hardware andsoftware is used to splice together the original recorded video and thesupplemental material to provide the final product, and generation ofthat final product using the video editing hardware and software mayrequire many hours of work by one or more video editing personnel.

SUMMARY

The present disclosure is directed to systems and methods for generatinga composite video based on a plurality of input streams. A first videostream is received from a first device. A second video stream andcoordination information are received from a second device. Thecoordination information includes instructions for combining the firstvideo stream and the second video stream in a composite video. The firstvideo stream and the second video stream are time aligned, and theinstructions of the coordination information and the time-aligned firstvideo stream and second video stream are used to generate the compositevideo.

In another method for generating a composite video based on a pluralityof input streams, a first video stream is received. A second videostream and coordination information are generated, where thecoordination information includes instructions for combining the firstvideo stream and the second video stream in a composite video. Thesecond video stream or the coordination information is generated basedon the received first video stream. The second video stream and thecoordination information are transmitted to a device configured tocombine the first video stream and the second video stream in acomposite video based on the coordination information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example computer-implemented system for generating acomposite video based on a first video stream and a second video stream.

FIG. 2 depicts an example composite video including a first video streamand a second video stream that may be generated with a multi-streamcompositing engine.

FIG. 3 depicts another computer-implemented system for generating acomposite video based on a first video stream and a second video stream.

FIG. 4 depicts different composite video configurations that may begenerated by a multi-stream compositing engine based on first and secondvideo streams and coordination information.

FIG. 5 depicts an example composite video display including a firstvideo stream containing an educational video and a second video streamcontaining supplemental presentation slides.

FIG. 6 depicts an example computer-implemented system for combining afirst video stream with a second video stream that includes annotationinformation.

FIG. 7 depicts an example computer-implemented system for combining afirst video stream with a second video stream that includes multipleindividual component streams.

FIG. 8 depicts an example computer-implemented system for combiningmultiple video streams in a composite video using a coordinating device.

FIG. 9 is a flowchart illustrating a computer-implemented method ofgenerating a composite video based on a plurality of input streams.

FIGS. 10A, 10B, and 10C depict example systems for use in implementing amulti-stream compositing engine.

DETAILED DESCRIPTION

FIG. 1 depicts an example computer-implemented system 100 for generatinga composite video 102 based on a first video stream 104 and a secondvideo stream 106. In the system 100, a first device 108 may transmit thefirst video stream 104 to a multi-stream compositing engine 110. Asecond device 112 may transmit the second video stream 106 andcoordination information 114 to the multi-stream compositing engine 110.The multi-stream compositing engine 110 may combine the first and secondvideo streams 104, 106 in the composite video 102 based on instructionsincluded in the coordination information 114. The instructions of thecoordination information 114 may be used to control, for example, therelative sizes and positions of the first and second video streams 104,106 in the composite video 102.

FIG. 2 depicts an example composite video 200 including a first videostream 202 and a second video stream 204 that may be generated with amulti-stream compositing engine. In the composite video 200, the firstvideo stream 202 includes video of a professor's lecture, and the secondvideo stream 204 includes related video intended to supplement astudent's understanding of the lecture. For example, after the professorexplains the four bases found in DNA in the first video stream 202, thesecond video stream 204 may be updated to display the image depicted inFIG. 2. The first and second video streams 202, 204 may be received fromseparate first and second devices and combined in the multi-streamcompositing engine based on instructions in received coordinationinformation from one or both of the devices.

In the example of FIG. 2, the coordination information may specifyparticular sub-regions of the overall composite video display 200 inwhich to display the first and second video streams 202, 204.Alternatively, the coordination information may specify the particularsub-regions while also providing instructions for controlling theplayback of the video streams 202, 204. For example, the coordinationinformation may instruct the multi-stream compositing engine to pausethe first video stream 202 at particular points to allow students totake notes. While first video stream 202 is paused, the coordinationinformation may allow the second video stream 204 to continue itsplayback. Numerous other video manipulation instructions may be includedin the coordination information (e.g., fast-forward, rewind, slowmotion). All such instructions may be reflected in the final compositevideo product 200, including the overall appearance of the display(i.e., the selected sub-regions of the display used for each stream) andthe temporal coordination of the two video streams' playback (i.e., thetime synchronization of streams 202, 204).

With reference again to FIG. 1, the second device 112 may be configuredto receive the first video stream 104. This may enable a user of thesecond device 112 to generate the second video stream 106 based on thefirst video stream 104. Thus, the user of the second device 112 mayrecord the second video stream 106 on the second device 112 whileviewing the first video stream 104. For example, if the second device112 includes a video camera component, the user may view the first videostream 104 on the device 112 and simultaneously record the second videostream 106 to respond to or supplement the first stream 104. In theexample of FIG. 2, the user of the second device 112 may view theprofessor's lecture in the first video stream 104 and choose the precisepoint in time in which to display the image containing the four DNAbases in the second video stream 106. When the first and second videostreams 104, 106 are time-aligned (i.e., synchronized) and combined inthe multi-stream compositing engine 110, the first and second videostreams 104, 106 may complement and/or supplement each other.

Numerous types of composite videos 102 may be generated in this manner.As in the previous examples, the first and second video streams 104, 106may each include full motion video footage. In another example, thesecond video stream 106 may include annotation information (i.e.,“virtual ink”) that may be “hand-drawn” over the first video stream 104and overlaid on top of it in the final composite video 102. In anotherexample, the second video 106 may include a still image (e.g., aphotograph, web browser screen capture, text display) represented as avideo stream. In another example, the second video stream 106 mayinclude no viewable material and may instead include only audio datarepresented as a video stream.

The receipt of the first video stream 104 at the second device 112 mayalso enable the user of the second device 112 to generate thecoordination information 114 based on the first video stream 104. In anexample, the coordination information 114 may be generated via asoftware program that allows the user to control and manipulate thevideo streams 104, 106 on a virtual “presentation canvas” to generatethe coordination information. With reference to the example of FIG. 2,the user may position the first and second video streams 202, 204 inparticular sub-regions of the presentation canvas software using a “dragand drop” technique. This positioning of the video streams 202, 204 maybe reflected in coordination information 114. After positioning thevideo streams 202, 204, the user may begin playback of the professor'slecture video in the first video stream 202. After beginning playback ofthe first stream 202, the user's subsequent actions within the virtualpresentation canvas software may be recorded and included in thecoordination information 114. For example, based on material presentedthroughout the professor's lecture video, the user may at various pointschange the video displayed in the second video stream 204. The videostreams 202, 204 may also be re-sized, slowed down, sped up, paused,stopped, rewound, or fast-forwarded. This manipulation of the secondvideo stream 204 may also be reflected in coordination information 114.

Thus, the user's drag and drop actions and video manipulation actionswithin the presentation canvas software program may be converted intomachine code instructions in the coordination information 114. Theseinstructions may be used by the multi-stream compositing engine 110 tocreate the composite video 102 according to the user's actions. Thecomposite video final product 102 may thus be a verbatim ornear-verbatim representation of the user's actions in the presentationcanvas software (i.e., the composite video 102 may reflect exactly whatthe user sees when coordinating the streams 104, 106 within the softwareof the second device 112). Using such software and the multi-streamcompositing engine 110, composite video 102 may be generated “on thefly” (i.e., the second video stream 106 may be generated and coordinatedwith the first video stream 104 during a live playback of the firststream 104). This may be an efficient method of producing the compositevideo 102.

Although expensive and time-consuming hardware and software videoediting techniques may be desirable for some applications (e.g.,producing a feature film or documentary), the “on the fly” productionmethod enabled by the system 100 may be desirable for many otherapplications. For example, as reflected in the example of FIG. 2,educational institutions may utilize the “on the fly” production methodto create composite video 200 by combining the professor's lecture videowith supplemental notes, presentation slides, websites, or other video.Because the system 100 may allow the composite video to be generated “onthe fly,” the system 100 may enable the composite video 200 to begenerated and distributed via a network (e.g., the Internet) in nearreal-time, such that students in remote locations could view the lectureand supplemental material at nearly the same time as the lecture isoccurring. For the near real-time distribution system to beaccomplished, the user of the second device 112 may receive a live feedof the first video stream 104 as it is occurring. The user maycoordinate the live feed of the first video stream 104 with thesupplemental second video stream 106 in the presentation canvassoftware. The multi-stream compositing engine 110 may receive live feedsof the video streams 104, 106 and coordination information 114 fromtheir respective sources as they are being generated. The multi-streamcompositing engine 110 may generate the composite video 102 from thelive feeds, allowing the composite video 102 to be distributed in nearreal-time.

Other potential applications may include live video commentary on a newsevent as it is occurring, a recorded board of directors meetinginvolving board members located in remote places, and audio commentaryand/or telestration annotation for a live sporting event. In anotherexample, the composite video 102 is not produced using live feeds of thefirst and second video streams 104, 106, and one or both of the videostreams 104, 106 may be pre-recorded and combined at a later time by themulti-stream compositing engine 110.

FIG. 3 depicts another computer-implemented system 300 for generating acomposite video 302 based on a first video stream 304 and a second videostream 306. Similar to FIG. 1, the system of FIG. 3 includes a firstdevice 308 configured to transmit the first video stream 304 to amulti-stream compositing engine 310. A second device 314 may transmitthe second video stream 306 to the multi-stream compositing engine 310.The first and second devices 308, 314 may interact with the multi-streamcompositing engine 310 through one or more servers 318, which can hostthe multi-stream compositing engine 310. The multi-stream compositingengine 310 may use the first and second video streams 304, 306 togenerate the composite video 302. The multi-stream compositing engine310 may be implemented using a processing system (e.g., one or morecomputer processors) executing software operations or routines forgenerating the composite video 302. User computers 314 (e.g., desktopcomputers, laptop computers, tablet computers, smart phones) mayinteract with the multi-stream compositing engine 310 through a numberof ways, such as over one or more networks 316. It should be understoodthat the multi-stream compositing engine 310 could also be provided on astand-alone computer for access by a user 314.

In the example system 300 of FIG. 3, the first device 308 or the seconddevice 314 may function as a coordinating device used to transmitcoordination information to the multi-stream compositing engine 310. Inanother example, both the first and second devices 308, 314 may beenabled to generate and transmit coordination information to themulti-stream compositing engine 310. In another example, one of thedevices 308, 314 may execute a full version of the virtual presentationcanvas software program described above, while the other device mayexecute a streamlined, “light” version of the software with fewerfeatures and/or editing capabilities.

FIG. 3 illustrates example components of the first and second devices308, 314 that may be used to produce the first and second video streams304, 306, respectively. The first device 308 may be an iPhone or othersmartphone and may include, for example, a video camera, still camera,audio recorder, and/or text generator 320. Video captured by the videocamera may be used to generate the first video stream 304 directly,while images, audio, and text generated by the still camera, audiorecorder, and text generator, respectively, may each be converted to avideo stream and placed in the first video stream 304. The second device314 may be an iPad or other tablet computing device and may include, forexample, a front camera, back camera, web browser, whiteboard softwareprogram, and annotation software program (e.g., a “virtual ink” ortelestrator program) 322. The front and back cameras may be used toproduce the second video stream 306 directly, while output from the webbrowser, whiteboard software program, and annotation software programmay be converted to a video stream and placed in the second video stream306. Other example second devices include, for example, a laptopcomputer, desktop computer, or smartphone device. The second device 314in this example may receive the first video stream 304 and allow for thecreation of the second video stream 306 via one or more of itscomponents 322 while the first video stream 304 is being played back.

In one example, the second device 314 may receive the first video stream308 from the one or more servers 318. Use of the one or more servers 318to deliver the first video stream 304 to the second device 314 may bedesirable in situations where a user of the second device 314 wishes tostop, pause, or rewind video of the first video stream 304 in thecomposite video 302. In these situations, the one or more servers 318may be configured to provide video buffering to the second device 314,such that the user can pause, stop, rewind, or otherwise manipulate alive feed of the first video stream 304 during creation of the compositevideo 302. In another example, the second device 314 may receive thefirst video stream 304 from an external network not connected to the oneor more servers 318 (e.g., a local area network, virtual privatenetwork, the Internet).

FIG. 4 depicts different composite video configurations that may begenerated by a multi-stream compositing engine based on first and secondvideo streams and coordination information. The different configurationsdepicted in FIG. 4 may be created, for example, using the examplecomputer-implemented systems of FIGS. 1 and 3, where a user of a seconddevice controls positioning and playback of first and second videostreams in the composite video using coordination information sent tothe multi-compositing engine. Display (a) depicts an example displaywhere the coordination information includes instructions to themulti-stream compositing engine to display only a first video stream(denoted throughout FIG. 4 in sub-regions of the composite video displaylabeled “1”) and to not display a second video stream (denotedthroughout FIG. 4 in sub-regions of the composite video display labeled“2”).

Display (b) of FIG. 4 depicts an example display where first and secondvideo streams are displayed adjacent to each other in separatesub-regions of the composite video display. Display (b) is similar tothe display depicted in FIG. 2, where the first video stream 202includes video of a professor's lecture and the second video stream 204includes supplemental video keyed to the content of the professor'slecture. The first and second video streams in display (b) may include,for example, video, a still image represented as a video stream, webbrowser output represented as a video stream, text represented as avideo stream, output from a whiteboard computer software programrepresented as a video stream, and output from presentation software(e.g., Microsoft PowerPoint, Adobe Acrobat) represented as a videostream.

Display (c) of FIG. 4 depicts an example display including a viewablefirst video stream and a second video stream that includes only audiodata represented as a video stream. Thus, the second video stream ofdisplay (c) includes no viewable video. Applications based on display(c) may include situations where one wishes to provide audio commentaryon a video-recorded event but does not wish to use any portion of theoverall composite video display for displaying video of a personproviding the commentary. Alternatively, displays similar to display (c)may be generated when the device generating the second stream does notinclude a video recording capability but allows for audio recording.

Display (d) of FIG. 4 depicts an example display including a large firstvideo stream overlaid with a smaller second video stream. An example useof display (d) may be where the second video stream includes annotationinformation (e.g., virtual ink) intended to be overlaid on top of thefirst video stream. For example, a user of the second device may use anannotation software program to “draw” over the first video stream.Alternatively, the overlaid second video stream may be used to implementa “picture-in-picture” display or a stock market ticker-tape displaythat covers a portion of the first video stream, for example.

Display (e) of FIG. 4 depicts an example display including a largerfirst video stream and multiple secondary video streams displayed inseparate sub-regions of the overall composite video display. An exampleuse of display (e) may be where a user wishes to supplement a primaryfirst video stream with multiple, different secondary video streams.With reference to FIG. 2, the user may wish to supplement theprofessor's lecture video stream 202 with both a still image and anumber of changing presentation slides. Using the presentation canvassoftware described above, the user may choose the particular sub-regionsof the composite video display in which to display the streams and maymanipulate and coordinate the playback of all streams. The display andcoordination decisions of the user may be reflected in instructions inthe coordination information used by the multi-stream compositingengine.

Display (f) of FIG. 4 depicts an example display where a first videostream is paused and a second video stream is playing. Display (f)illustrates that coordination information generated by a second devicemay instruct the multi-stream compositing engine to change the playbackproperties of the first and second video streams, such that one or bothof them is not played continuously. Other standard video manipulationtechniques may be similarly enabled and reflected in the composite video(e.g., fast-forwarding, rewinding, stopping, slow motion).

Display (g) of FIG. 4 depicts an example display where a second videostream may include instant replay video of a portion of a first videostream. Thus, in this example, a live playback of the first video streamis minimized in size, and the instant replay video is displayed in alarger sub-region of the composite video display.

FIG. 5 depicts an example composite video display 500 including a firstvideo stream 502 containing an educational video 502 and a second videostream 504 containing supplemental presentation slides. Composite video500 is an example composite video 500 formatted similarly to display (b)of FIG. 4. In FIG. 5, first video stream 502 may be output from a firstdevice and delivered to a multi-stream compositing engine and a seconddevice. The second device may be configured to execute a softwareprogram for creating the composite video 500 (e.g., software thatutilizes a virtual “presentation canvas” for manipulating andcoordinating the first video stream 502 and the second video stream504). A user of the software program may choose a size for the firstvideo stream 502 and position it within a particular sub-region of theoverall composite video display 500. The user may further introduce thedepicted presentation slides to be output as the second video stream504. While viewing a live feed of the first video stream 502 or playingback a pre-recorded version of it, the user may advance through variouspresentation slides in the second video stream 504. The layout of thefirst and second video streams 502, 504 and the particular temporalcoordination between the streams 502, 504 may be reflected incoordination information sent to the multi-stream compositing engine.The multi-stream compositing engine may use the received video streams502, 504 and the coordination information to produce the composite videodisplay 500 according to what a user sees when manipulating andcoordinating the videos 502, 504 within the presentation canvas program.Because the first and second video streams may be recordedasynchronously, the multi-stream compositing program may perform a timealignment (i.e., synchronization) procedure on the first and secondvideo streams 502, 504 to ensure proper temporal coordination in theplayback of the video streams 502, 504 in the composite video 500.

FIG. 6 depicts an example computer-implemented system 600 for combininga first video stream 602 with a second video stream 604 that includesannotation information. In FIG. 6, a first device 606 may transmit thefirst video stream 602 to a multi-stream compositing engine 608. Thefirst device 606 may be, for example, a mobile device configured torecord video (e.g., an iPhone, iPad, or other similar mobile computingdevice), and the first video stream 602 may include full motion videorecorded using the first device 606. A second device 610 may transmitthe second video stream 604 and coordination information 612 to themulti-stream compositing engine 608.

In the example of FIG. 6, the second video stream 604 includesannotation information (i.e., “virtual ink” created by allowing a userof the second device 610 to “draw” over the first video stream 602 as itis being played back within a presentation canvas software program).Although the user may view the first video stream 602 while creating thesecond video stream 604, the second video stream 604 may contain onlynew information added to the first video stream 602. In the example ofFIG. 6, the new information added to the first video stream 602 includesonly the virtual ink lines, and thus, only this new information isincluded in the second video stream 604 transmitted to the multi-streamcompositing engine 608. The transmission of only the new information inthe second video stream 604 may be used to avoid unnecessaryre-transmission of material contained within the first video stream 602.This may be particularly desirable in situations where network bandwidthis at a premium and/or when the second device 610 is configured toreceive only a lower bitrate version of the first video stream 602.

The multi-stream compositing engine 608 may combine the first and secondvideo streams 602, 604 in the composite video output 614 usinginstructions in the coordination information 612. In the example of FIG.6, the coordination information 612 may instruct the multi-streamcompositing engine 608 to display both the first and second videostreams 602, 604 full-screen and to overlay the second video stream 604over the first video stream 602 to enable the desired annotation effect.

FIG. 7 depicts an example computer-implemented system 700 for combininga first video stream 702 with a second video stream 704 that includesmultiple individual component streams 706, 708. In FIG. 7, a firstdevice 710 may transmit the first video stream 702 to a multi-streamcompositing engine 712 and to a second device 714. Based on the firstvideo stream 702, a user of the second device 714 may generate thecomponent video streams 706, 708 to be transmitted to the multi-streamcompositing engine 712 and combined with the first video stream 702.Coordination information 716 may also be transmitted from the seconddevice 714 to the multi-stream compositing engine 712 to instruct themulti-stream compositing engine 712 as to how the video streams 702, 704should be combined in composite video 718.

In the example of FIG. 7, the coordination information 716 may containinstructions regarding the sizing and positioning of the first videostream 702 and of the component video streams 706, 708 comprising thesecond video stream 704. Thus, in the example composite video 718, thefirst video stream 702 is reproduced at a larger size than the componentvideo streams 706, 708 of the second video stream 704. As explainedabove, sizing and temporal coordination instructions of the coordinationinformation 716 may be created using a software program that utilizes avirtual presentation canvas, which may allow the user to drag, drop,re-size, and manipulate (e.g., fast-forward, rewind, pause, stop) thevarious video streams 702, 704 to dictate an appearance of the compositevideo 718.

FIG. 8 depicts an example computer-implemented system 800 for combiningmultiple video streams 802 in a composite video 804 using a coordinatingdevice 810. The multiple video streams 802 may be created using one ormore video capture devices 808 and transmitted to a multi-streamcompositing engine 806. The multiple streams 802 may then be transmittedfrom the multi-stream compositing engine 806 to the coordinating device810 via a network connection 812. The coordinating device 810 may allowa user of the coordinating device 810 to determine how the multiplestreams 802 are to be combined in the composite video 804. Theinstructions for combining the streams 802 may be included incoordination information 814, which may be transmitted from thecoordinating device 810 to the multi-stream compositing engine 806. Inthe example of FIG. 8, the coordinating device 810 may not generate avideo stream and may be used only for generating the coordinationinformation 814 (e.g., using a virtual presentation canvas softwareprogram).

FIG. 9 is a flowchart 900 illustrating a computer-implemented method ofgenerating a composite video based on a plurality of input streams. At902, a first video stream is received from a first device. At 904, asecond video stream and coordination information are received from asecond device. The coordination information includes instructions forcombining the first video stream and the second video stream in acomposite video. At 906, the first video stream and the second videostream are time-aligned. The time alignment procedure may allow thefirst and second video streams to be synchronized in the compositevideo. At 908, the instructions of the coordination information and thetime-aligned first and second video streams are used to generate thecomposite video.

The concepts of this disclosure have been described in the context ofexamples, and the scope of the disclosure includes many additionalexamples. For example, certain multi-stream composite video generationfunctionality may be provided to one or more devices as a downloadableapplication (e.g., for free or for a fee). In one configuration, acomposite video creation session may be set up by a host personoperating a downloadable application on a host video capture device(e.g., the second device 112 of FIG. 1, which may include thepresentation canvas software program). The host video capture device(e.g., an iPad or other tablet computing device) may be used to sendinvitations to client participants via applications (e.g., the same hostapplication or a client application) operating on their client videocapture devices. Upon initiations of the session (e.g., upon sending ofthe invitations, at an identified start time, upon a signal from thehost), the host video capture device and the client video capturedevices may be free to begin recording input video streams. The hostvideo capture device may allow a user to view the input video streams ofthe client video capture devices and to coordinate the appearance of thecomposite video using the presentation canvas software while recording avideo input stream. Upon completion of recording, the client videocapture devices may transmit their input video streams to a multi-streamcompositing engine (e.g., based on a user command, automatically uponclose of an application, based on a signal from a host).

As another example, once a composite video has been created, it can beshared via e-mail, text message, or social media posting (e.g.,Facebook, MySpace, Twitter, YouTube, StreamWeaver). In anotherconfiguration, a composite video can be broadcast live via a website,social media outlet, or otherwise, where each of the input video streamstransmits live to the multi-stream compositing engine, which constructsand transmits/streams the composite video live.

As a further example, FIGS. 10A, 10B, and 10C depict example systems foruse in implementing a multi-stream compositing engine. For example, FIG.10A depicts an exemplary system 1200 that includes a standalone computerarchitecture where a processing system 1202 (e.g., one or more computerprocessors located in a given computer or in multiple computers that maybe separate and distinct from one another) includes a multi-streamcompositing engine 1204 being executed on it. The processing system 1202has access to a computer-readable memory 1206 in addition to one or moredata stores 1208. The one or more data stores 1208 may include inputstreams 1210 as well as coordination information 1212.

FIG. 10B depicts a system 1220 that includes a client serverarchitecture. One or more user PCs 1222 access one or more servers 1224running a multi-stream compositing engine 1226 on a processing system1227 via one or more networks 1228. The one or more servers 1224 mayaccess a computer readable memory 1230 as well as one or more datastores 1232. The one or more data stores 1232 may contain input streams1234 as well as coordination information 1236.

FIG. 10C shows a block diagram of exemplary hardware for a standalonecomputer architecture 1250, such as the architecture depicted in FIG.10A that may be used to contain and/or implement the programinstructions of system embodiments of the present invention. A bus 1252may serve as the information highway interconnecting the otherillustrated components of the hardware. A processing system 1254 labeledCPU (central processing unit) (e.g., one or more computer processors ata given computer or at multiple computers), may perform calculations andlogic operations required to execute a program. A non-transitoryprocessor-readable storage medium, such as read only memory (ROM) 1256and random access memory (RAM) 1258, may be in communication with theprocessing system 1254 and may contain one or more programminginstructions for performing the method of generating a composite video.Optionally, program instructions may be stored on a non-transitorycomputer readable storage medium such as a magnetic disk, optical disk,recordable memory device, flash memory, or other physical storagemedium.

A disk controller 1260 interfaces one or more optional disk drives tothe system bus 1252. These disk drives may be external or internalfloppy disk drives such as 1262, external or internal CD-ROM, CD-R,CD-RW or DVD drives such as 1264, or external or internal hard drives1266. As indicated previously, these various disk drives and diskcontrollers are optional devices.

Each of the element managers, real-time data buffer, conveyors, fileinput processor, database index shared access memory loader, referencedata buffer and data managers may include a software application storedin one or more of the disk drives connected to the disk controller 1260,the ROM 1256 and/or the RAM 1258. Preferably, the processor 1254 mayaccess each component as required.

A display interface 1268 may permit information from the bus 1252 to bedisplayed on a display 1270 in audio, graphic, or alphanumeric format.Communication with external devices may optionally occur using variouscommunication ports 1272.

In addition to the standard computer-type components, the hardware mayalso include data input devices, such as a keyboard 1273, or other inputdevice 1274, such as a microphone, remote control, pointer, mouse and/orjoystick.

Additionally, the methods and systems described herein may beimplemented on many different types of processing devices by programcode comprising program instructions that are executable by the deviceprocessing subsystem. The software program instructions may includesource code, object code, machine code, or any other stored data that isoperable to cause a processing system to perform the methods andoperations described herein and may be provided in any suitable languagesuch as C, C++, JAVA, for example, or any other suitable programminglanguage. Other implementations may also be used, however, such asfirmware or even appropriately designed hardware configured to carry outthe methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, datainput, data output, intermediate data results, final data results, etc.)may be stored and implemented in one or more different types ofcomputer-implemented data stores, such as different types of storagedevices and programming constructs (e.g., RAM, ROM, Flash memory, flatfiles, databases, programming data structures, programming variables,IF-THEN (or similar type) statement constructs, etc.). It is noted thatdata structures describe formats for use in organizing and storing datain databases, programs, memory, or other computer-readable media for useby a computer program.

The computer components, software modules, functions, data stores anddata structures described herein may be connected directly or indirectlyto each other in order to allow the flow of data needed for theiroperations. It is also noted that a module or processor includes but isnot limited to a unit of code that performs a software operation, andcan be implemented for example as a subroutine unit of code, or as asoftware function unit of code, or as an object (as in anobject-oriented paradigm), or as an applet, or in a computer scriptlanguage, or as another type of computer code. The software componentsand/or functionality may be located on a single computer or distributedacross multiple computers depending upon the situation at hand.

While the disclosure has been described in detail and with reference tospecific embodiments thereof, it will be apparent to one skilled in theart that various changes and modifications can be made therein withoutdeparting from the spirit and scope of the embodiments. Thus, it isintended that the present disclosure cover the modifications andvariations of this disclosure provided they come within the scope of theappended claims and their equivalents.

It should be understood that as used in the description herein andthroughout the claims that follow, the meaning of “a,” “an,” and “the”includes plural reference unless the context clearly dictates otherwise.Also, as used in the description herein and throughout the claims thatfollow, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise. Further, as used in the description hereinand throughout the claims that follow, the meaning of “each” does notrequire “each and every” unless the context clearly dictates otherwise.Finally, as used in the description herein and throughout the claimsthat follow, the meanings of “and” and “or” include both the conjunctiveand disjunctive and may be used interchangeably unless the contextexpressly dictates otherwise; the phrase “exclusive of may be used toindicate situations where only the disjunctive meaning may apply.

It is claimed:
 1. A computer-implemented method for generating compositevideo instructions, the method comprising: receiving, at a coordinatingdevice, a first video stream from a first device and a second videostream from a second device; displaying, on the coordinating device, thefirst video stream, the second video stream, and supplemental content,wherein the supplemental content comprises text, images, or video;detecting user interactions with the coordinating device controlling anaspect of how the first video stream, the second video stream, or thesupplemental content is displayed on the coordinating device; detectinguser interactions with the coordinating device indicating a selection tonot display at least one of the first video stream or the second videostream generating, at the coordinating device, coordination instructionsbased on the received user interactions, wherein the coordinationinstructions includes a command for controlling the aspect of how thefirst video stream, the second video stream, or the supplemental contentis displayed in a composite video, and wherein the coordinationinstructions further includes an indication to not display the selectedfirst video stream or second video stream; and transmitting thecoordination instructions to a compositing engine distinct from thecoordinating device, wherein the compositing engine uses thecoordination instructions for combining the first video stream, thesecond video stream, and the supplemental content into the compositevideo according to the aspect of how the first video stream, the secondvideo stream, or the supplemental content is displayed and theindication to not display the selected first video stream or secondvideo stream.
 2. The method of claim 1 wherein the supplemental contentcomprises a website, a presentation, or a third video stream.
 3. Themethod of claim 1 wherein the supplemental content is received by thecoordinating device from a third device.
 4. The method of claim 1wherein the aspect is the position of the first video stream, the secondvideo stream, or the supplemental content on the display.
 5. The methodof claim 4 wherein the user interactions controlling the position of thefirst video stream, the second video stream, or the supplemental contentcomprises a drag and drop action.
 6. The method of claim 1 wherein theaspect is the displayed size of the first video stream, the second videostram, or the supplemental content.
 7. The method of claim 1 wherein theaspect is playback instructions controlling the playback of the firstvideo stream, the second video stream, or the supplemental content.
 8. Anon-transitory computer-readable medium encoded with instruction that,when executed by a processor, perform a method for generating compositevideo instructions, the method comprising: receiving, at a coordinatingdevice, a first video stream from a first device; displaying, on thecoordinating device, the first video stream and supplemental content,wherein the supplemental content comprises text, images, or video;detecting user interactions with the coordinating device controlling anaspect of how the first video stream or supplemental content isdisplayed on the coordinating device; generating, at the coordinatingdevice, coordination instructions based on the received userinteractions, wherein the coordination instructions includes a commandfor controlling the aspect of how the first video stream or thesupplemental content is displayed in a composite video; and transmittingthe coordination instructions to a compositing engine distinct from thecoordinating device, wherein the compositing engine uses thecoordination instructions for combining the first video stream and thesupplemental content into the composite video according to the aspect ofhow the first video stream or the supplemental content is displayed. 9.The non-transitory computer-readable medium of claim 8 wherein thesupplemental content comprises a website, a presentation, or a secondvideo stream.
 10. The non-transitory computer-readable medium of claim 8wherein the supplemental content is received by the coordinating devicefrom a third device.
 11. The non-transitory computer-readable medium ofclaim 8 wherein the aspect is the position of the first video stream orthe supplemental content on the display.
 12. The non-transitorycomputer-readable medium of claim 11 wherein the user interactionscontrolling the position of the first video stream or the supplementalcontent comprises a drag and drop action.
 13. The non-transitorycomputer-readable medium of claim 8 wherein the aspect is the displayedsize of the first video stream or the supplemental content.
 14. Thenon-transitory computer-readable medium of claim 8 wherein the aspect isplayback instructions controlling the playback of the first video streamor the supplemental content.